Comparability of Computerized Adaptive and Paper-Pencil Tests

When a traditional Paper-Pencil Test (PPT) is delivered by computer, two types of computerization can be implemented. One is a linear Computer-Based Test (CBT) in which the paper version of the test is presented and administered via computers. In a linear CBT, the items on both versions are identical, in general, and scoring methods and procedures are the same. The change from PPT to CBT, therefore, only involves the change of administration mode. The other type of computerization is the Computerized Adaptive Testing (CAT) in which not only the medium of administration changes from paper to computer but also the test delivery algorithm turns from linear to adaptive. This adaptive testing paradigm allows the test items to be selected and administered so that they are tailored to each test taker’s ability. Therefore, in comparability studies, both the administration mode and paradigm effect on examinees’ performance should be examined to ensure the comparability of the CAT and its PPT counterpart. The administration mode and paradigm effect on examinees’ performance should be examined to ensure the comparability of the CAT and its PPT counterpart. Paradigm Effects The administration mode effect has been widely examined in the comparison of PPT and linear CBT. Although findings are not conclusive, there seems to be a trend indicating that the two versions are comparable across the administration mode (e.g. Paek, 2005, Wang, Jiao, Young, Brooks, & Olson 2007, 2008). When CAT is compared to its PPT counterpart, the mode effect and paradigm effect are confounded with each other. In order to separate the two effects and examine the paradigm effect, some studies have focused on comparability analysis between the linear CBT and CAT.